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Air-to-Air  Missile  Testing  Using  Advanced  Distributed  Simulation 

By  Dr.  LarryMcKee 

Science  Applications  International  Corporation,  JADS  JTF,  Albuquerque,  New  Mexico 

BACKGROUND 

The  Joint  Advanced  Distributed  Simulation  Joint  Test  and  Evaluation  (JADS  JT&E)  was 
chartered  by  the  Deputy  Director,  Test,  Systems  Engineering  and  Evaluation  (Test  and 
Evaluation),  Office  of  the  Under  Secretary  of  Defense  (Acquisition  and  Technology)  in  October 
1994  to  investigate  the  utility  of  Advanced  Distributed  Simulation  (ADS)  technologies  for 
support  of  Developmental  Test  and  Evaluation  (DT&E)  and  Operational  Test  and  Evaluation 
(OT&E).  The  program  is  Air  Force  led,  with  Army  and  Navy  participation,  and  is  nominally 
scheduled  for  five  years. 

The  JADS  JT&E  is  tasked  in  three  areas: 

Investigate  tie  present  utility  of  ADS  for  T&E. 

Identify  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E. 
-  Identify  the  requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to  support 
a  more  complete  T&E  capability  in  the  future. 

The  JADS  JT&E  is  executing  its  charter  by  evaluating  the  application  of  ADS  to  the  T&E  of 
representative  military  systems.  The  types  of  systems  selected  represent  three  slices  of  the  T&E 
spectrum:  a  System  Integration  Test  (SIT)  which  explores  ADS  support  of  air-to-air  missile 
testing,  an  End-To-End  (ETE)  test  which  explores  ADS  support  for  Command,  Control, 
Communications,  Computers,  and  Intelligence  (C4I)  testing,  and  an  Electronic  Warfare  (EW)  test 
which  explores  ADS  support  for  EW  testing.  The  JTF  will  broaden  the  conclusions  developed  in 
the  three  dedicated  test  areas  by  leveraging  off  of  ADS  activities  sponsored  and  conducted  by 
other  agencies. 

The  SIT  consists  of  two  phases.  The  first  phase,  the  Linked  Simulators  Phase  (LSP),  has  been 
completed  and  employed  an  all-simulator  architecture  for  testing  an  AIM-9  Sidewinder  missile. 
The  second  phase  is  now  underway  and  links  live  shooter  and  target  aircraft  to  an  AIM-120 
AMRAAM  hardware-in-the-loop  (HWIL)  simulation  at  Eglin  AFB. 

LSP  OVERVIEW 

The  LSP  was  executed  by  the  JADS  Joint  Test  Force  (JTF)  and  the  Naval  Air  Warfare  Center, 
Weapons  Division  (NAWCWPNS)  between  August  and  November  1996.  The  SIT  missions 
simulate  a  single  shooter  aircraft  launching  an  air-to-air  missile  against  a  single  target  aircraft. 
The  scenario  utilized  in  the  LSP  missions  was  taken  from  previous  AIM-9M  testing  and  is  shown 
in  Figure  1.  This  scenario  was  replicated  during  LSP  testing. 
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•  10,400  ft/ 0.72  mach 

•  58  0  angle  off  tail 

•  3.6  g  level  turn 

•  flare  countermeasures 

Figure  1.  AIM-9M-8/9  Live  Fire  Profile  (LPN-15,  9  June  93) 

In  the  LSP,  the  shooter,  target,  and  missile  were  all  represented  by  simulators.  ADS  techniques 
were  used  to  link  NAWCWPNS  manned  flight  laboratories  representing  the  aircraft  to  an  air-to- 
air  missile  hardware-in-the-loop  (HWIL)  laboratory  representing  the  missile.  The  LSP  test 
configuration  is  shown  in  Figure  2.  The  F/A-18  Weapon  System  Support  Facility  (WSSF)  at 
China  Lake  and  the  F-14D  Weapon  System  Integration  Center  (WSIC)  at  Point  Mugu  were  the 
shooter  and  target,  respectively.  These  laboratories  were  linked  to  each  other  and  to  an  AIM- 
9M-8/9  FtWIL  laboratory  at  the  Simulation  Laboratory  (SIMLAB)  at  China  Lake.  Interfacing  of 
the  laboratories  to  the  network  was  by  means  of  network  interface  units  (NIUs).  The  launch 
aircraft  laboratory  “fired”  the  AIM-9  in  the  SIMLAB  at  the  simulated  target  aircraft,  and  the 
AIM-9  seeker  responded  to  infrared  (IR)  sources  in  the  SIMLAB  which  simulated  the  IR 
signatures  and  relative  motions  of  the  target  aircraft  and  the  flare  countermeasures.  Real-time 
links  between  the  laboratories  allowed  the  players  to  respond  to  each  other. 

The  nodes  exchanged  entity  state  information  with  each  other  by  means  of  Distributed  Interactive 
Simulation  protocol  data  units  (DIS  PDUs).  However,  the  Stores  Management  System  (SMS) 
data  exchange  between  the  F/A-18  WSSF  and  the  AIM-9  SIMLAB  used  the  tactical  MIL-STD- 
1553  protocol,  because  no  suitable  DIS  protocol  exists  for  these  data,  because  this  exchange  was 
only  between  the  WSSF  and  the  SIMLAB,  and  because  use  of  the  tactical  protocol  was 
appropriate  for  integrated  weapon  system  testing. 

The  test  runs  were  controlled  by  either  the  Battle  Management  Interoperability  Center  (BMIC)  at 
Point  Mugu  or  the  Test  Control  and  Analysis  Center  (TCAC)  in  Albuquerque.  The  control  center 
ensured  that  all  nodes  were  ready  for  each  run  and  issued  the  commands  to  start  and  stop  the 
runs.  PDUs  were  processed  at  the  control  center  to  provide  JADS  test  controllers  and  analysts 
with  real-time  stealth  node  viewing  of  the  simulated  engagement. 
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Figure  2.  Linked  Simulators  Phase  Test  Configuration 

LSP  TEST  PLAN 

The  LSP  was  designed  to  examine  the  relationships  between  network  performance,  system  under 
test  (i.e.,  AIM-9)  data,  and  test  measures  of  interest.  The  test  objectives  were: 

Objective  1:  Assess  the  validity  of  AIM-9  data  obtained  in  the  LSP  ADS  configuration 
Objective  2:  Assess  utility  of  LSP  ADS  configuration  for  parametric  studies 
Objective  3:  Assess  effect  of  latency  on  validity  of  test  results 
Objective  4:  Assess  ability  of  LSP  ADS  configuration  to  support  AIM-9  testing 
(This  test  objective  was  broken  into  subobjectives  as  follows.) 

Subobjective  4-1:  Assess  capability  of  network  to  provide  required  bandwidth  and 
connectivity 

Subobjective  4-2:  Assess  the  effects  of  ADS-induced  errors  on  LSP  test  results  validity 
Subobjective  4-3:  Assess  adequacy  of  standard  data  protocols  for  LSP  test 
Subobjective  4-4:  Assess  reliability,  availability,  and  maintainability  of  ADS  network 
Subobjective  4-5:  Assess  capability  for  centralized  test  control  and  monitoring 

Three  formal  testing  periods,  or  missions,  were  planned. 
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V&V  Mission:  the  objective  was  to  validate  the  performance  of  the  LSP  ADS  configuration 
by  replicating  the  live  test  engagement  (Fig.  1)  and  comparing  results  from  the  LSP  to  results 
from  the  live  test. 

Parametric  Study  Mission:  the  objective  was  to  evaluate  the  ability  to  repeat  the  baseline 
scenario  with  controlled  variations  in  a  single  parameter.  The  parameters  to  be  varied  related 
to  countermeasure  effectiveness  and  were  to  be  the  timing  of  flare  release  and/or  target 
evasive  maneuver  relative  to  missile  launch.  Two  test  methods  were  to  be  used.  In  the 
manual  method,  the  aircraft  lab  pilots  were  to  fly  the  desired  parametric  case.  In  the 
automatic  replay  method,  the  output  from  the  aircraft  labs  during  one  of  the  manual  runs  was 
to  be  replayed  and  used  to  drive  the  missile  HWIL  simulation. 

Latency  Study  Mission:  the  objective  was  to  incrementally  increase  the  latency  between  the 
WSIC  and  the  SIMLAB  and  to  note  the  corresponding  effect  on  the  missile  performance. 
This  technique  was  to  be  used  to  determine  the  maximum  allowable  latency  for  which  the 
various  nodes  still  agree  on  the  engagement  conditions  and  outcome. 

The  schedule  planned  for  executing  these  missions  is  shown  in  the  upper  part  of  Figure  3. 
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Figure  3.  Linked  Simulators  Phase  Test  Schedule 


LSP  TEST  RESULTS 

The  bottom  part  of  Figure  3  shows  the  time  lines  for  actual  testing.  The  time  required  to 
integrate  and  check  out  the  architecture  was  significantly  longer  than  planned.  Full-up 
architecture  configurations  were  required  to  verify  fixes  to  integration  problems.  The  periods  of 
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integrated  verification  work  were  called  “lab  times”,  and  they  moved  scheduled  test  activity  to  the 
right  on  the  timeline.  As  a  result  of  the  slips,  the  Latency  Mission  was  not  performed.  However, 
the  JTF  collected  sufficient  data  from  the  other  testing  to  evaluate  the  latency  test  objective. 

The  results  of  the  test  execution  is  as  follows: 

-  Testing  was  performed  during  four  half-day  periods  (the  Test  Rehearsal  was  done  over 
two  days),  and  160  runs  were  attempted.  About  two  thirds  of  these  (107  runs)  were 
successfully  linked  runs  with  complete  missile  flyouts. 

-  Some  of  the  causes  of  the  aborted  or  incomplete  runs  were  procedural  problems  (e.g.,  labs 
not  reset  for  run),  hardware  or  software  failures  (e.g.,  NIU  failure  or  lab  not  sending  out 
PDUs),  unsuccessful  missile  launch  (due  to  shooter  losing  lock  before  launch  or  hung 
missile),  and  the  SIMLAB  flight  table  exceeding  its  limits  (which  terminated  the  missile 
flyout). 

Overall,  the  LSP  testing  configuration  had  a  good  availability  of  about  85%. 

A  number  of  problems  were  encountered  during  the  testing. 

-  The  automatic  replay  runs  did  not  work  as  planned.  The  objective  of  these  runs  was  to 
achieve  a  precise  replication  of  one  of  the  manual  runs.  However,  this  could  not  be  done 
due  to  several  manual  actions  required  for  the  automatic  replay  runs:  manual  start  of  the 
aircraft  laboratories  replay,  manual  trigger  squeeze  by  the  shooter  pilot,  and  manual 
initiation  of  the  flare  simulation  in  the  SIMLAB.  This  problem  was  due  to  the 
fundamental  design  of  the  simulation  laboratories  used  and  could  not  be  fixed  without 
significant  cost  and  schedule  slips.  Because  of  these  problems,  the  automatic  replay  runs 
were  eliminated  from  the  Parametric  Study  Mission. 

-  The  latency  exhibited  large  values  and  sample-to-sample  variations  during  a  run.  In  early 
testing,  large  instantaneous  latency  values  (“spikes”)  were  observed,  some  in  excess  of  a 
second.  This  problem  was  solved  in  later  testing  by  resetting  the  NIUs  before  each  run. 
There  were  several  problems  in  interfacing  the  shooter  and  target  data  to  the  missile 
simulation  in  the  SIMLAB. 

—  The  missile  exhibited  lofting  in  which  it  gained  altitude  upon  launch  before  guiding  to 
the  target.  This  was  invalid  missile  behavior  (the  shooter  was  at  a  higher  altitude  than 
the  target).  The  problem  was  due  to  improper  conversion  of  the  shooter  vertical 
velocity  to  the  SIMLAB  reference  frame  and  was  fixed  during  testing. 

—  The  target  position  as  determined  by  the  missile  simulation  steady  diverged  from  the 
true  target  location  during  the  missile  flyout.  This  effect  was  significantly  reduced 
during  testing  by  correcting  the  SIMLAB  coordinate  frame  conversion  and  by 
increasing  the  target  velocity  update  rate. 

—  The  target  location  in  the  SIMLAB  reference  frame  was  improperly  initialized.  This 
caused  an  improper  presentation  of  the  target  trajectory  to  the  missile.  As  a  result,  the 
missile  was  successfully  intercepting  the  target  in  the  SIMLAB  reference  frame,  but 
external  observers  saw  the  missile  miss  the  target  by  over  1000  ft.  This  problem  was 
not  discovered  until  after  testing  was  over,  but  has  since  been  fixed. 
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The  results  of  evaluating  the  test  objectives  are  as  follows. 

-  The  SIMLAB  missile  performance  was  valid  for  its  target  representation.  However,  its 
target  representation  was  not  completely  valid. 

The  aircraft  laboratory  pilots  were  able  to  achieve  good  manual  reproducibility  of  the 
engagement  scenario.  This  supports  the  conclusion  that  the  LSP  configuration  has  some 
utility  for  parametric  studies. 

-  Latency  was  small  enough  in  later  trials  such  that  the  different  nodes  could  agree  on 
launch  conditions  to  within  10%  of  the  shot  box.  However,  latency  was  large  and  variable 
enough  such  that  the  target  and  missile  could  disagree  on  whether  or  not  the  target  was 
“killed.” 

-  The  ADS  network  provided  adequate  bandwidth. 

There  were  no  significant  ADS-induced  errors. 

-  The  DIS  PDUs  used  were  adequate  for  entity  state  data  exchanges. 

There  was  good  availability  of  the  LSP  testing  configuration  and  no  wide  area  network 
failures. 

-  The  centralized  test  control  procedures  worked  well. 

LSP  UTILITY  CONCLUSIONS 

The  results  of  LSP  testing  support  the  following  conclusions  on  the  utility  of  the  LSP  ADS 
configuration. 

-  The  LSP  ADS  configuration  has  utility  for  missile  weapon/launch  aircraft  system  T&E 
—  The  configuration  successfully  ran  integrated  scenarios/profiles  among  linked 

laboratories. 

—  This  configuration  can  be  used  for  discrepancy/deficiency  resolution,  especially  when 
there  are  interface  issues/problems  between/among  weapon  systems  (e.g.,  the  aircraft 
radar,  mission  computer,  stores  management  system,  and  the  missile).  This  includes 
troubleshooting  problems  which  prove  to  be  difficult  to  replicate,  particularly  those 
that  appear  in  flight  tests  but  are  not  readily  duplicated  in  stand-alone  laboratory 
testing. 

—  Linked  laboratories  permit  the  HWIL  missile  to  respond  to  actual  pre-  and  post-launch 
weapon  system  inputs,  instead  of  relying  on  stand-alone  “canned”  inputs,  in  a  more 
operationally  realistic  environment 

-  The  LSP  ADS  configuration  has  utility  for  parametric  studies  involving  a  one-on-one  air- 
to-air  missile  engagement. 

—  The  key  characteristic  of  a  parametric  study  is  the  ability  to  repeat  a  given  scenario 
with  either  no  changes  or  with  a  single  parameter  varying. 

—  The  manual  method  for  replicating  a  given  profile  resulted  in  very  good  run-to-run 
reproducibility  of  the  launch  conditions. 

—  The  automatic  replay  method,  as  implemented  in  the  LSP,  was  unable  to  precisely 
replicate  a  given  scenario  and  had  no  advantage  over  the  manual  method. 
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-  The  LSP  ADS  configuration  has  utility  for  rehearsal  and  refinement  of  live  engagement 
scenarios. 

—  Pilot  training  and  rehearsals  of  live  missile  firings  requiring  difficult  and/or  precise 
launch  conditions  could  be  accomplished  using  this  configuration.  The  ADS  link 
could  be  used  by  the  pilot  to  practice  (pre-fly)  the  mission  in  a  controlled  laboratory 
environment  before  using  aircraft  and  range  assets.  ADS  could  assist  in  doing  the 
flight  test  right  the  first  time  which  translates  into  reduced  aircraft  flight  hours,  range 
time,  etc. 

-  The  LSP  ADS  configuration  has  utility  for  terminal  engagement  studies  involving  open- 
loop  interactions  between  the  missile  and  the  target  (missile  responds  to  target,  but  target 
does  not  respond  to  missile). 

-  Latency  does  not  affect  these  results,  as  long  as  the  latency  values  are  relatively 
constant  during  a  run. 

-  The  LSP  ADS  configuration  does  not  have  utility  for  terminal  engagement  studies 
involving  closed-loop  interactions  between  the  missile  and  the  target  (missile  and  target 
respond  to  each  other). 

—  Target  position  errors  and  latencies  were  such  that  the  nodes  disagreed  on  the  final 
range  between  the  missile  and  the  target  by  more  than  the  missile  lethal  radius.  Hence, 
the  nodes  could  disagree  on  whether  or  not  the  target  had  been  “killed.” 

LESSONS  LEARNED 

-  The  network  interface  to  the  simulations  is  critical 

—  Accurate  coordinate  transformations  are  necessary.  These  proved  difficult  at  the 
beginning  of  the  LSP.  Coordinate  transformations  must  be  verified  and  validated  at 
each  site  and  then  reverified  and  revalidated  during  end-to-end  testing  as  early  as 
possible  in  the  test  phase.  Personnel  who  are  subject  matter  experts  in  coordinate 
transformations  must  be  assigned  and  readily  available  during  this  process. 

—  The  NIUs  need  improvement.  NIUs  of  some  sort  are  necessary  if  two  simulators 
cannot  communicate  directly  in  a  common  language,  and  on  a  common  timeline. 
NIUs  can  be  a  major  source  of  both  error  and  processing  delays.  For  the  LSP,  the 
NIUs  were  difficult  to  troubleshoot  and  control.  Future  projects  should  use  an 
improved  DIS  NIU  or  an  “NIU  function”  (in  the  master  simulation  computer)  which 
provides  a  more  direct  user  control  of  the  content  of  the  data  and  network 
communications,  including  the  capability  to  force  network  communications  at  a  user- 
specified  frame  rate.  Such  improvements  could  simplify  the  overall  network/ADS 
configuration,  as  well  as  the  troubleshooting  and  resolution  of  various 
network/ADS/DIS  problems. 

-  Common  ADS-related  hardware  and  software  is  needed.  It  was  difficult  to  get  the  ADS 
network  to  behave  in  a  uniform  fashion  due  to  the  many  different  types  of  interface 
hardware,  communications  equipment  (routers),  and  interface  software  versions. 
Additionally,  lack  of  common  software  resulted  in  the  NIUs  having  numerous  and 
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different  problems  related  to  conversions,  timing,  CPU  speed,  etc.  Whenever  possible  for 
future  ADS  test,  the  network  hardware  and  interface  software  should  be  exactly  the  same 
among  all  the  sites. 

Latency  variations  were  significant.  Aggregate  latency  includes  the  transmission  delays 
and  the  processing  delays  on  each  leg  of  the  ADS  architecture.  Both  the  transmission- 
induced  and  processing  components  of  latency  exhibited  significant  random  variations 
which  cannot  be  compensated  for,  although  the  processing  delays  were  the  dominant 
source  of  latency.  Future  ADS  implementations  which  require  low  latencies  (e.g., 
interactive  missile  engagement  analyses)  should  focus  on  techniques  for  reducing  the 
latency  between  the  network  interface  and  the  receiving  simulations. 

Time  sources  must  be  synchronized.  Time  sources  at  the  various  sites  must  be 
synchronized  off  of  the  same  master  time  source  and  then  must  be  validated  at  each  test 
site  prior  to  project  operations  to  ensure  accurate,  synchronized  time  is  precisely  recorded 
at  each  test  site.  It  took  a  lot  of  effort  to  get  clock  synchronization  values  into  the  few 
millisecond  region,  and  meaningful  latency  measurements  were  impossible  without  this 
degree  of  clock  synchronization. 

Quantitative  validation  has  limitations.  The  JTF  intent  was  to  quantitatively  verify  missile 
simulation  performance  against  live  fire  data.  Given  the  facts  that  only  a  single  live  fly 
event  was  available  to  support  the  process  and  that  the  live  engagement  could  not  be 
perfectly  replicated,  it  became  necessary  to  modify  the  validation  approach.  The  modified 
approach  included  both  qualitative  (in  which  missile  trajectory  shapes  were  compared)  and 
quantitative  methods  and  successfully  identified  invalid  results  (lofting  missile  trajectories 
and  target  initialization  errors). 

A  stepped  buildup  approach  should  be  used 

—  Systematic  checkout  of  the  standalone  simulators  is  needed  before  linking,  especially 
verification  of  simulation  laboratory  modifications  required  for  ADS  linking.  The 
modifications  should  be  performed  early  in  the  buildup  and  carefully  checked  and 
verified  before  linked  testing. 

—  Direct  (non-DIS)  links  should  be  used  during  test  buildup.  This  focuses  early 
verification  checks  on  making  the  linked  architecture  work  without  the  additional 
interfaces  and  reference  frame  transformations  needed  for  DIS  implementation.  This 
also  provides  a  benchmark  for  the  DIS  implementation 

—  Structured  testing  of  the  network  must  be  performed  prior  to,  and  independent  of,  the 
linked  testing  times  and  the  simulation  laboratories  to  validate  transmission/reception 
rates,  bandwidth  utilization,  latency,  data  transmission  and  reception,  etc.  prior  to 
commencing  project  test  periods.  A  “test,  analyze,  fix,  test”  approach  in  combination 
with  structured,  independent  testing  of  the  network  during  the  LSP  would  have  been 
beneficial.  In  several  cases  during  the  LSP,  linked  testing  time  was  used  for  testing  the 
network  where  independent  network  testing  would  have  been  more  cost  effective. 


8 


Special  test  equipment  is  needed  for  checkout  and  verification.  Special  test  equipment  and 
other  networking  tools  should  be  part  of  each  simulation  node’s  configuration  during  the 
development,  test,  checkout,  and  verification/validation  phases  in  all  subsequent  ADS 
testing.  Special  test  equipment  and  networking  tools  are  required  to  more  rapidly  isolate 
and  determine  the  specific  cause  of  network,  ADS/DIS,  etc.  problems.  Without  the 
special  test  equipment  and/or  tools,  trial  and  error  becomes  the  normal  troubleshooting 
mode  which  increases  the  resource  requirements  (time,  schedule,  cost,  etc.).  The 
equipment  and  tools  also  permit  off-line  testing  without  the  exclusive  scheduling  of 
simulation  laboratories  and  the  associated  costs.  Individual  test  sites  could  then  check 
their  own  software  and  hardware,  verify  PDUs  and  other  ADS/DIS  requirements,  and 
verify  program-specific  requirements  prior  to  the  more  costly  linked  tests.  The  special  test 
equipment  and  networking  tools  developed  and  used  by  the  JADS  JTF  for  the  LSP  proved 
valuable  once  the  equipment  and  tools  became  available  to  NAWCWPNS. 

Linking  of  facilities  using  ADS  can  require  significant  facility  interface  hardware  and 
software  development.  ADS  implementation  is  not  “plug  and  play.”  Significant  checkout 
activities  were  required  using  the  full-up  linked  architecture. 

Additional  time  is  needed  before  the  beginning  and  after  the  end  of  each  testing  period. 
Allocate  a  minimum  of  an  additional  two  hours  of  laboratory  time  at  the  end  of  each  test 
period  for  data  logging,  data  archiving,  data  transfer,  and  laboratory  reclassification. 
Allocate  a  minimum  of  an  additional  one  hour  of  laboratory  set-up  time  prior  to  each  test 
period.  These  were  the  normal  pre-  and  post-test  laboratory  times  required  for  each  LSP 
formal  test  period.  The  pre-  and  post-test  requirements  should  be  included  in  the  number 
of  laboratory  hours  needed  for  each  test  period  and  incorporated  into  the  planned  costs. 

Effective  data  management  is  needed.  Linked  laboratories  can  generate  a  large  volume  of 
data  at  distributed  locations.  Without  careful  planning,  key  data  may  not  be  collected 
and/or  transmitted  to  the  analysis  center,  and  data  collected  at  the  sites  may  not  be  in  a 
useful  form  for  centralized  analysis.  A  comprehensive  data  management  plan  must  be 
developed  before  testing  which  clearly  identifies  (1)  the  data  to  be  collected  at  each  site, 
(2)  on-site  processing  of  the  data,  and  (3)  data  to  be  transmitted  to  the  analysis  center. 

Adequate  time  must  be  allotted  for  data  analysis  between  test  events.  There  was  a 
tendency  to  underestimate  the  time  required  to  adequately  analyze  the  large  volume  of 
data  collected  in  the  test  events.  As  a  result,  some  problems  from  one  mission  were  not 
fully  diagnosed  and  fixed  before  the  next  mission.  In  fact,  some  problems  (e.g.,  target 
initialization  errors)  were  not  even  recognized  until  all  testing  was  over.  Rehearsal  of  the 
analysis  procedures  should  be  used  to  better  estimate  the  time  required  for  adequate 
analysis  between  test  events. 

Test  communications  requirements  must  be  addressed  early  in  the  test  planning  phase 
—  This  is  necessary  to  ensure  effective  communications  during  the  test.  Remote  test 
control  using  two  non-secure  telephone  conference  bridges  (i.e.,  two  communications 
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nets)  was  acceptable.  However,  the  audio  level  between  connections  varied,  making 
“loud  and  clear”  communications  among  all  the  sites  difficult  at  times 

—  A  standard,  linked  test  should  have  multiple  (more  than  two)  communications  nets 
(e.g.,  control,  analyst,  network,  and  internal  laboratory)  with  easy,  selectable  access  to 
all  the  nets  from  multiple  locations  within  the  laboratory.  A  minimum  of  one  secure 
telephone  at  each  site  is  also  required.  More  complex,  linked  tests  may  require 
additional  non-secure  and/or  secure  communications  nets.  Recommend  laboratories 
research  various  communication  options  by  reviewing  their  local  range  control  center 
and  telemetry  data  center  communication  setups. 

—  The  capability  for  secure  video  teleconferences  (VTCs)  among  multiple  (more  than 
two)  sites  is  required  when  planning,  coordinating,  and/or  briefing  integrated  weapon 
system  tests  among  various  sites. 

Configuration  control  is  essential.  Configuration  control  of  the  network  and  the  ADS/DIS 
system,  including  its  hardware,  software,  and  its  simulator  interfaces,  is  necessary  starting 
at  the  beginning  of  the  program.  This  includes  a  “scientific”  approach  to  network 
management  and  troubleshooting.  Either  a  single  person  or  a  network  committee  should 
be  in  charge  of  the  configuration  control/network  management.  The  level  of  control  will 
vary  with  the  phase  of  the  project.  Since  problems  are  part  of  the  process,  the  network 
configuration  cannot  be  “frozen”  until  there  is  an  agreed  upon  “baseline.”  However,  the 
configuration  control  process/procedures,  individual/committee  in  charge,  etc.  must  be 
established  at  the  beginning  of  the  program  and  followed  until  the  end  of  the  program. 

SUT  experts  must  be  involved  from  the  beginning.  Weapon  system(s)  analysis  experts 
must  be  planned  for  (and  budgeted  for)  to  analyze  the  weapon  system-related  data  from 
the  system  under  test  (SUT)  and  to  provide  the  analytical  results,  conclusions,  and 
recommendations.  There  should  be  more  than  one  expert,  and  they  must  be  involved  from 
the  beginning  of  the  project  to  establish  the  data  and  instrumentation  requirements, 
verify /validate  the  analytical  approach,  assist  in  the  development  of  test  matrices  and  test 
procedures,  and  provide  overall  weapon  system  expertise.  Options  include  the  support 
and/or  user  agencies  providing  the  SUT  analysis  experts.  Both  agencies  could  provide 
their  own  experts  who  would  independently  analyze  the  data  from  the  standpoint  of  the 
test  objectives.  The  independent  analyses  would  then  be  compared,  and  the  SUT  experts 
would  resolve  any  differences  in  their  conclusions. 

Future  ADS  T&E  projects  should  be  conducted  following  established  T&E  flight  test 
practices  and  procedures.  The  LSP  was  more  typical  of  a  T&E  flight  test  effort  than  a 
standard  laboratory  test.  Specifically: 

—  Standard  Universal  Documentation  System  (UDS)  documents  such  as  requirements 
documentation  (e.g.,  Program  Introduction  (PI)  and/or  Operation  Requirements  (OR)) 
from  the  user  agency  and  response  documentation  (e.g.,  Statement  of  Capability 
(SOC)  and/or  Operations  Directive  (OD))  from  the  support  agency  should  be  used. 
This  would  establish  a  clear  set  of  requirements  at  the  beginning  of  the  program  from 
the  user  agency  and  a  clear  statement  of  the  support  agency’s  capabilities,  constraints, 
and  limitations  in  meeting  those  requirements. 
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Test  monitoring/control  and  test  procedures/conduct  should  be  run  similar  to  a  flight 
test.  Detailed  test  cards  should  be  drafted,  reviewed,  distributed,  briefed,  and  used  for 
each  mission.  Back-up  test  cards  should  also  be  considered  and  briefed  for 
contingency  purposes.  Briefs  and  debriefs  should  be  conducted  before  and  after  each 
mission.  The  briefs  should  cover  such  items  as  the  test  objectives;  telephone 
numbers/frequencies  to  use  for  test  control,  etc.;  test  configuration  of  each  laboratory; 
instrumentation  and  data  collection  requirements;  go/no  go  criteria;  contingency/  back¬ 
up  plans;  test  conduct  including  a  detailed  review  of  the  test  cards;  communications 
procedures;  OPSEC;  and  the  time  and  place  of  the  debrief.  A  briefing  checklist  should 
be  developed  and  used.  The  LSP  used  one  basic  profile  which  permitted  simplified 
test  cards  and  test  procedures.  These  simplified  cards  and  procedures,  including  the 
use  of  a  few  “step”  calls,  were  satisfactory  for  the  LSP. 
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